How groups make good decisions

Enlarge

Whether a committee is choosing strategies, a swarm of bees is searching for a nest site, or multiple AI agents are determining a solution – many decisions are made in a group rather than alone. But which procedures lead to good results, especially for complex problems with many possible answers? LMU philosopher Professor Christian List and economist Professor Franz Dietrich from the Paris School of Economics address this topic in a study published in Philosophical Transactions of the Royal Society B.

“The study looks at the conditions under which groups arrive at coherent and reliable collective judgments,” explains Christian List, Chair of Philosophy and Decision Theory at LMU and co-director of the Munich Center for Mathematical Philosophy. “We wanted to ascertain which methods of aggregation are particularly suitable for this purpose – that is, how individual judgments can be combined so as to produce sound collective decisions.”

Classic example of collective decision-making

This applies not only to binary decisions like the “guilty” or “not guilty” verdicts of a jury trial, but also to estimates made on the basis of many possible values, such as macroeconomic or meteorological variables, probabilities of economic or geopolitical events, or travel directions. Decisions are also made collectively in animal groups, for instance when choosing a flight direction or grazing spot.

List cites a classic example of collective decision-making through aggregation from 1907: “At an agricultural fair, the English statistician Francis Galton evaluated the results of a contest in which participants of the fair were asked to guess the weight of an ox.” The almost 800 submitted estimates varied widely. “Yet the median – that is, the value that lies exactly in the middle when the estimates are aligned from lowest to highest – came surprisingly close to the ox’s actual weight.” List explains that the median value can be reliable despite the huge variation in individual estimates, provided the estimates are independent of one another and have a sufficient probability of neither overestimating nor underestimating the correct value.

“But collective decision making also raises fundamental challenges,” observes List. “For one thing, even coherent individual opinions can lead to contradictory overall results when several related questions are evaluated simultaneously. For example, a group may judge by a majority that A is the case, and that B necessarily follows from A, and yet conclude that B is not the case: that is, reach a contradictory overall result.

Clearly, a decision-making body ought to avoid inconsistent results. An expert panel, for example, would lack credibility, not to mention utility, if it failed to produce a coherent overall result. Furthermore, decision procedures can be manipulated through strategic voting, such as when individuals deliberately misrepresent their opinions to steer the outcome in a preferred direction. A central question also concerns the conditions under which a procedure can help increase the likelihood of factually correct collective judgments.

Clear rules

The researchers translated such problems into a formal – i.e., mathematically defined – model that specifies which questions and answers are possible and which combinations of them logically fit together. On this basis, they derived general statements about the strengths and limitations of various aggregation methods. Using mathematical theorems, they then compared various methods of aggregation: such as determining the median as per Galton’s method, or the average, or applying the majority rule for binary questions which can be answered with “yes” or “no.” In the latter case, the option that receives more than half the votes in a group for an individual question is deemed to have prevailed.

“A key finding of our study is that good collective decisions are certainly possible, but they require clear rules and a precise understanding of their limitations,” said List. Collective intelligence does not arise automatically from many individuals being involved. “What is crucial, rather, is the rules by which their judgments are aggregated and how the questions to be assessed are structured.”

The median rule performed comparatively well. “It is more robust to outliers, harder for participants to manipulate strategically, and, under certain assumptions, good at delivering accurate results,” says List. However, the median is not a universal solution, but performs best under favorable initial conditions. “If the respondents – such as the participants at the agricultural fair – possess sufficient expertise and their judgments are independent of one another, the probability of correct collective judgments using the median method increases as the group size grows.” Equally, the study makes clear that these favorable conditions are often only partially fulfilled in practice. “In many cases, individuals rely on similar information, influence one another, or react to the same external factors.”

From climate forecasts to AI agents

At the same time, the results show that when dealing with complex, interrelated problems, key requirements for aggregation methods – such as consistency, truth-tracking, and protection against strategic manipulation – are often not fully compatible with one another: we can fulfill only some of these requirements but often not all. In the study, the authors discuss ways of addressing these tensions, such as restricting the admissible inputs in their model or using methods in which judgments on key questions are first aggregated and further values are inferred from them.

List explains that their findings – including those from their larger research project on judgment aggregation, from which the newly published study emerged – are relevant to many fields, ranging from expert panels to economic and climate forecasts, and even legal and political decision-making processes. This is because they help select, for different contexts, procedures that lead to the most reliable collective decisions possible. “In addition, they are significant for artificial intelligence systems in which multiple agents aggregate information or data sources.”

Publication: 

Christian List, Franz Dietrich: “Collective Intelligence through Aggregation”. In: Philosophical Transactions of the Royal Society B (doi.org/10.1098/rstb.2024.0454)